
<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

    <title>HTTP proxies &#8212; LAVA 2024.05 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="_static/doctools.js"></script>
    <script src="_static/sphinx_highlight.js"></script>
    <link rel="shortcut icon" href="_static/favicon.ico"/>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Administering VLANd support in LAVA" href="vland-admin.html" />
    <link rel="prev" title="Deploying an iPXE (x86) device" href="ipxe.html" />
    <link rel="canonical" href="https://docs.lavasoftware.org/lava/proxy.html" />
  
<meta charset='utf-8'>
<meta http-equiv='X-UA-Compatible' content='IE=edge,chrome=1'>
<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1'>
<meta name="apple-mobile-web-app-capable" content="yes">
<script type="text/javascript" src="_static/js/jquery-1.12.4.min.js"></script>
<script type="text/javascript" src="_static/js/jquery-fix.js"></script>
<script type="text/javascript" src="_static/bootstrap-3.4.1/js/bootstrap.min.js"></script>
<script type="text/javascript" src="_static/bootstrap-sphinx.js"></script>


  </head><body>

  <div id="navbar" class="navbar navbar-default navbar-fixed-top">
    <div class="container">
      <div class="navbar-header">
        <!-- .btn-navbar is used as the toggle for collapsed navbar content -->
        <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
        </button>
        <a class="navbar-brand" href="index.html"><span><img src="_static/lava.png"></span>
          LAVA</a>
        <span class="navbar-text navbar-version pull-left"><b>2024.05</b></span>
      </div>

        <div class="collapse navbar-collapse nav-collapse">
          <ul class="nav navbar-nav">
            
                <li><a href="genindex.html">Index</a></li>
                <li><a href="contents.html">Contents</a></li>
            
            
              <li class="dropdown globaltoc-container">
  <a role="button"
     id="dLabelGlobalToc"
     data-toggle="dropdown"
     data-target="#"
     href="index.html">Site <b class="caret"></b></a>
  <ul class="dropdown-menu globaltoc"
      role="menu"
      aria-labelledby="dLabelGlobalToc"><ul class="current">
<li class="toctree-l1"><a class="reference internal" href="index.html">Introduction to LAVA</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="contents.html">Contents</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="glossary.html">Glossary of terms</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="support.html">Getting support</a></li>
</ul>
</ul>
</li>
              
                <li class="dropdown">
  <a role="button"
     id="dLabelLocalToc"
     data-toggle="dropdown"
     data-target="#"
     href="#">Page <b class="caret"></b></a>
  <ul class="dropdown-menu localtoc"
      role="menu"
      aria-labelledby="dLabelLocalToc"><ul>
<li><a class="reference internal" href="#">HTTP proxies</a></li>
<li><a class="reference internal" href="#using-the-http-proxy">Using the HTTP proxy</a></li>
<li><a class="reference internal" href="#using-the-http-cache">Using the HTTP cache</a></li>
<li><a class="reference internal" href="#handling-bots">Handling bots</a></li>
</ul>
</ul>
</li>
              
            
            
              
                
  <li>
    <a href="ipxe.html" title="Previous Chapter: Deploying an iPXE (x86) device"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">&laquo; Deploying an ...</span>
    </a>
  </li>
  <li>
    <a href="vland-admin.html" title="Next Chapter: Administering VLANd support in LAVA"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Administering... &raquo;</span>
    </a>
  </li>
              
            
            
            
            
              <li class="hidden-sm"></li>
            
          </ul>

          
            
<form class="navbar-form navbar-right" action="search.html" method="get">
 <div class="form-group">
  <input type="text" name="q" class="form-control" placeholder="Search" />
 </div>
  <input type="hidden" name="check_keywords" value="yes" />
  <input type="hidden" name="area" value="default" />
</form>
          
        </div>
    </div>
  </div>

<div class="container">
  <div class="row">
    <div class="body col-md-12 content" role="main">
      
  <section id="http-proxies">
<span id="proxy"></span><h1>HTTP proxies<a class="headerlink" href="#http-proxies" title="Permalink to this heading">¶</a></h1>
<p>When running jobs, LAVA is fetching many resources from remote servers
over http.
In some situation or when many jobs are running in parallel, the network
performances could become a bottleneck.</p>
<p>To improve network performances, admins could setup a caching service that will
keep a local version of the resources used by LAVA.</p>
<p>Admin can choice among two kind of caching service:</p>
<ul class="simple">
<li><p>generic http proxy like <a class="reference external" href="http://www.squid-cache.org">squid</a></p></li>
<li><p>specific http cache like <a class="reference external" href="https://cache.lavasoftware.org/">KissCache</a></p></li>
</ul>
</section>
<section id="using-the-http-proxy">
<h1>Using the HTTP proxy<a class="headerlink" href="#using-the-http-proxy" title="Permalink to this heading">¶</a></h1>
<p>The dispatcher will use the proxy configured in the HTTP_PROXY environment
variable.</p>
<p>Environment variables are set in:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">/etc/lava-server/env.yaml</span></code> for every dispatchers</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/etc/lava-server/dispatcher.d/&lt;name&gt;/env.yaml</span></code> for a specific dispatcher</p></li>
</ul>
</section>
<section id="using-the-http-cache">
<h1>Using the HTTP cache<a class="headerlink" href="#using-the-http-cache" title="Permalink to this heading">¶</a></h1>
<p>The dispatcher will use the caching service defined in the dispatcher
configuration in <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/dispatcher.d/&lt;name&gt;/dispatcher.yaml</span></code>.</p>
<p>Set <code class="docutils literal notranslate"><span class="pre">http_url_format_string</span></code> to the url of the local caching service.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">http_url_format_string</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;https://cache.lavasoftware.org/api/v1/fetch?url=%s&quot;</span>
</pre></div>
</div>
</section>
<section id="handling-bots">
<h1>Handling bots<a class="headerlink" href="#handling-bots" title="Permalink to this heading">¶</a></h1>
<p>LAVA has a lot of URLs which may take a lot of work to render on the server. If
automated search bots routinely try to fetch these URLs, the instance can have
performance issues.</p>
<p>LAVA includes a default <code class="docutils literal notranslate"><span class="pre">robots.txt</span></code> template which disallows the dynamic
content to reduce the impact on the server. <code class="docutils literal notranslate"><span class="pre">static/</span></code> is allowed so that
files like the documentation can be indexed.</p>
<p>To serve a custom <code class="docutils literal notranslate"><span class="pre">robots.txt</span></code> from the root of the instance, using Apache,
add an alias at the top of <code class="docutils literal notranslate"><span class="pre">/etc/apache2/sites-available/lava-server.conf</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Alias</span> <span class="o">/</span><span class="n">robots</span><span class="o">.</span><span class="n">txt</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">share</span><span class="o">/</span><span class="n">lava</span><span class="o">-</span><span class="n">server</span><span class="o">/</span><span class="n">static</span><span class="o">/</span><span class="n">robots</span><span class="o">.</span><span class="n">txt</span>
</pre></div>
</div>
<p>Some bots will handle <code class="docutils literal notranslate"><span class="pre">/robots.txt`</span> <span class="pre">but</span> <span class="pre">some</span> <span class="pre">do</span> <span class="pre">not</span> <span class="pre">(or</span> <span class="pre">mishandle</span> <span class="pre">options</span>
<span class="pre">within</span> <span class="pre">the</span> <span class="pre">file).</span> <span class="pre">To</span> <span class="pre">handle</span> <span class="pre">this,</span> <span class="pre">django</span> <span class="pre">supports</span> <span class="pre">``DISALLOWED_USER_AGENTS</span></code>
and this is exposed in <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code>.</p>
<blockquote>
<div><p>“DISALLOWED_USER_AGENTS”: [“yandex”, “bing”],</p>
</div></blockquote>
<p>The values in DISALLOWED_USER_AGENTS are translated into regular expressions
using <code class="docutils literal notranslate"><span class="pre">re.compile</span></code> with <code class="docutils literal notranslate"><span class="pre">re.IGNORECASE</span></code> so are case-insensitive.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Always check your <code class="docutils literal notranslate"><span class="pre">robots.txt</span></code> and <code class="docutils literal notranslate"><span class="pre">DISALLOWED_USER_AGENTS</span></code>
settings on a local VM before modifying the main instance.</p>
</div>
</section>


    </div>
      
  </div>
</div>
<footer class="footer">
  <div class="container">
    <p class="pull-right">
      <a href="#">Back to top</a>
      
    </p>
    <p>
        &copy; Copyright 2010-2019, Linaro Limited.<br/>
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br/>
    </p>
  </div>
</footer>
  </body>
</html>